Hybrid depth detection and movement detection

ABSTRACT

A head-mounted device (HMD) is configured to perform depth detection in conjunction with movement tracking. The HMD includes a stereo camera pair comprising a first camera and a second camera, both of which are mounted on the HMD. The fields of view for both of the cameras overlap to form an overlapping field of view. These cameras are configured to detect both visible light and infrared (IR) light. The HMD also includes an IR dot-pattern illuminator that is configured to emit an IR dot-pattern illumination. The HMD uses the IR dot-pattern illumination to determine an object&#39;s depth. The HMD also includes one or more flood IR light illuminators that emit a flood of IR light. The HMD uses the flood of IR light to track at least its own movements, and sometimes even hand movements, in various environments, even low light environments.

BACKGROUND

Mixed-reality systems, including virtual-reality and augmented-realitysystems, have received significant attention because of their ability tocreate truly unique experiences for their users. For reference,conventional virtual-reality (VR) systems create a completely immersiveexperience by restricting their users' view to only a virtualenvironment. This is often achieved through the use of a head-mounteddevice (HMD) that completely blocks any view of the real world. As aresult, a user is entirely immersed within the virtual environment. Incontrast, conventional augmented-reality (AR) systems create anaugmented-reality experience by visually presenting virtual objects thatat least partially occlude portions of the real world.

As used herein, VR and AR systems are described and referencedinterchangeably. Unless stated otherwise, the descriptions herein applyequally to all types of mixed-reality systems, which (as detailed above)includes AR systems, VR systems, and/or any other similar system capableof displaying virtual objects.

The disclosed mixed-reality systems use one or more on-body devices(e.g., the HMD, a handheld device, etc.). The HMD provides a displaythat enables a user to view overlapping and/or integrated visualinformation in whatever environment the user is in, be it a VRenvironment or an AR environment. By way of example, as shown in FIG. 1,a mixed-reality system may present virtual content to a user in the formof a simulated vase resting on a real table surface.

Continued advances in hardware capabilities and rendering technologieshave greatly improved how mixed-reality systems render virtual objects.However, the process of immersing a user into a mixed-realityenvironment creates many challenges, difficulties, and costs,particularly with regard to determining depth and tracking movement.

For instance, by way of example, conventional depth detection systemsfail to adequately determine the depth of a smooth surface (e.g., awall) in a mixed-reality environment because those systems fail toadequately distinguish one part of the smooth surface from another part.Furthermore, conventional head tracking systems fail to adequately trackthe movements of a HMD in a low light environment because those systemsfail to adequately identify a sufficient number of anchor points. Assuch, there is a substantial need to improve how depth is detected,especially for smooth surfaced objects in mixed-reality environments.

Additionally, conventional HMD systems require separate/additionalhardware for performing depth detection from the hardware that isrequired to perform head tracking. This additional hardware adds to theoverall cost, weight, battery consumption and size of the HMD systems.

The subject matter claimed herein is not limited to embodiments thatsolve any disadvantages or that operate only in environments such asthose described above. Rather, this background is provided only toillustrate one exemplary technology area where some embodimentsdescribed herein may be practiced.

BRIEF SUMMARY

Disclosed embodiments include methods and systems incorporatinghead-mounted devices (HMD) with a stereo camera system in which the HMDsare configured to perform depth detection in conjunction with movementdetection. The cameras are mounted on the HMD and are able to obtainvisible light and IR light images. At least a part of the cameras'fields of view overlap. The HMD also includes one (or more) IRdot-pattern illuminators that emit an IR dot-pattern illumination whichis used to determine depth. The HMD also includes one or more flood IRlight illuminators that emit a flood of IR light so the HMD can trackmovement in a low light environment.

In some embodiments camera-based movement tracking is performed inconjunction with operations for performing depth detection. At a firstfrequency, the flood IR light illuminator emits a flood of IR light intoa low light environment. The HMD obtains images of reflected IR lightand then uses these images to track its movements of the HMD. At asecond frequency, the dot-pattern IR illuminator emits a dot-pattern IRlight illumination which can beneficially add texture to the environmentfor performing depth detection. In particular, reflected IR light isused during depth detection to construct geometric surfaces in theenvironment surrounding the HMD.

This Summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used as an aid in determining the scope of the claimed subjectmatter.

Additional features and advantages will be set forth in the descriptionwhich follows, and in part will be obvious from the description, or maybe learned by the practice of the teachings herein. Features andadvantages of the disclosed embodiments may be realized and obtained bymeans of the instruments and combinations particularly pointed out inthe appended claims. Features of the disclosed embodiments will becomemore fully apparent from the following description and appended claimsor may be learned by the practice of the embodiments as set forthhereinafter.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to describe the manner in which the above-recited and otheradvantages and features can be obtained, a more particular descriptionof the subject matter briefly described above will be rendered byreference to specific embodiments which are illustrated in the appendeddrawings. Understanding that these drawings depict only typicalembodiments and are not therefore to be considered to be limiting inscope, embodiments will be described and explained with additionalspecificity and detail through the use of the accompanying drawings inwhich:

FIG. 1 shows a head-mounted device (HMD) structured to identify itslocation and orientation with respect to its surrounding environment(i.e. motion detection) as well as to determine the depth of an objectin that environment. FIG. 1 also illustrates a table and a virtualobject (i.e. the vase) that are visible to the user of the HMD.

FIG. 2 illustrates a HMD that includes a stereo camera pair which can beused to perform motion detection and which can also be used to performdepth detection using an overlapping field of view region existingbetween the two cameras' fields of view.

FIG. 3 shows an example environment in which the HMD may be used, andthis example environment includes some objects that havetextureless/smooth surfaces.

FIG. 4 shows an infrared (IR) dot-pattern illuminator and an IRdot-pattern illumination emitted by the IR dot-pattern illuminator.

FIG. 5A shows the IR dot-pattern illumination projected into a HMD'ssurrounding environment, and FIG. 5B shows an example of how the camerasmay be oriented in relation to the IR dot-pattern illuminator and theiroverlapping fields of view.

FIG. 6A shows an environment as viewed in the visible light spectrumwhere two objects are present while FIG. 6B shows the same environmentas viewed in the IR light spectrum where the same two objects are beingilluminated by visible light and the IR dot-pattern illumination.

FIG. 7 demonstrates how the IR dot-pattern illuminator may be placed onthe HMD and aimed in a particular manner so that its IR dot-patternillumination will overlap the two cameras' overlapping field of viewregion.

FIG. 8 shows a depth map result of performing depth detection using thedisclosed principles in which one object is properly segmented fromanother object and in which depth values are given to the objects,regardless of whether they include textureless/smooth surfaces.

FIG. 9 illustrates various non-limiting position configurations forpositioning an IR dot-pattern illuminator on a HMD.

FIG. 10 illustrates various different integrated head tracking and depthdetector computer system components.

FIG. 11 provides an example method that may be performed with stereohead tracking cameras to determine a surface's depth, even if thatsurface is textureless/smooth, where the depth is detected using bothvisible light and IR light.

FIG. 12 shows another example method for performing depth detection withstereo head tracking cameras and which is performed in conjunction withmovement detection.

FIGS. 13A and 13B show how a flood IR illuminator may be used to projecta flood of IR light onto an environment in order to detect anchor pointsin a low light environment.

FIG. 14 shows an example configuration of the regions of IR lightassociated with flood IR illuminators mounted on a HMD, as well as thefields of view of the stereo cameras mounted on the HMD.

FIG. 15 shows various different head tracking computer systemcomponents.

FIG. 16 illustrates an example method for performing movement detectionin various environments, including low visible light environments.

FIG. 17 illustrates an example method for adjusting an illuminationintensity of a flood IR illuminator mounted to a HMD.

FIG. 18 depicts a hybrid HMD that includes both an IR dot-patternilluminator as well as a flood IR illuminator.

FIG. 19 shows a hybrid of head tracking (i.e. movement detection) anddepth detector computer system components.

FIG. 20 provides an example method for performing movement detection ina low light environment in conjunction with depth detection.

FIG. 21 illustrates an example computer system that may be used toperform embodiments disclosed herein.

DETAILED DESCRIPTION

At least some of the embodiments described herein relate to head-mounteddevices (HMD) configured to perform depth detection and movementdetection. The HMD may include a stereo camera pair comprising a firstand second camera which are both mounted on the HMD and which are bothable to detect visible light and infrared (IR) light. Such positioningis beneficial for motion tracking purposes because it allows the camerasto capture a large amount of area in the surrounding environment. Bycapturing more area, the HMD is better able to track movements.Furthermore, at least a part of the cameras' fields of view overlap withone another.

The HMD also includes one (or more) IR dot-pattern illuminators. The IRdot-pattern illuminator emits an IR dot-pattern illumination that spansan illumination area. The HMD uses the IR dot-pattern illumination todetermine depth for at least a part of the environment. The HMD alsoincludes one or more flood IR light illuminators that emit a flood of IRlight. This process improves how movement is tracked, particularly in alow light visible environment. The IR dot-pattern illuminator emits itsIR light according to a first frequency while the flood IR lightilluminator emits its IR light according to a second frequency.

In some embodiments camera-based movement tracking is performed withreflected flood IR light at a first frequency by the HMD while depthdetection processing is performed by the HMD with reflected dot-patternIR light at a second frequency. For instance, at a first frequency, theflood IR light illuminator emits a flood of IR light into a low lightenvironment. The head tracking stereo camera pair then obtains one ormore images of reflected IR light. The HMD uses these images to thentrack its own movements. At a second frequency, the dot-pattern IRilluminator emits a dot-pattern IR light illumination onto at least apart of the low light environment to add predetermined texture to theHMD's environment. Reflected IR light is then detected by the HMD's headtracking stereo camera pair. Using this reflected IR light, the HMD isable to construct a geometric surface for at least a part of theenvironment.

The movement tracking processes and the depth detection processes can beperformed at separate times/intervals or concurrently/with overlappingintervals.

It will be appreciated that the disclosed embodiments providesignificant improvements over how HMDs perform depth detection,particularly for detecting the depth of objects having smooth surfaces,and movement detection, particularly in low visible light environments.For instance, in at least some instances, the disclosed embodimentsprovide improvements for determining depth of objects within theenvironment surrounding a HMD, even objects having smooth surfaces. Thisis accomplished, for example, by using the HMD to apply and sense IRdot-pattern texturing applied to the surfaces of the objects in theenvironment surrounding the HMD. In this manner, it is possible toclearly and accurately determine any object's depth, even when thatobject has smooth surfaces. The disclosed embodiments also provideimprovements for tracking movement within the HMD's environment, evenwhen that environment is a low visible light environment. For instance,by projecting a flood of IR light into the environment, it is possibleto use IR light to track movement instead of using only visible light totrack movement.

Additionally, the present embodiments repurpose some of the hardwareused to perform head tracking, hand tracking, and depth detection (i.e.,the stereo camera system) to additionally perform depth detection andhand tracking to thereby eliminate some of the undesired costs, weight,battery consumption and size of alternate HMD systems (e.g., time offlight hardware) that would otherwise be required to enable a HMD toperform both head tracking, hand tracking, and depth detection.

Having just described some of the various high-level features andbenefits of the disclosed embodiments, attention will now be directed toFIGS. 1 through 12. These figures illustrate various architectures,methods, and supporting illustrations related to adding texture to asurface to better determine that surface's depth. Following thatdiscussion, the disclosure will turn to FIGS. 13A through 17. Thesefigures present various architectures, methods, and supportingillustrations related to projecting a flood of IR light into anenvironment to better perform movement detection (including headmovements, hand movements, hand-held device movements, etc.). Performingthis movement detection may be performed in any kind of environment,even in a low light environment. Subsequently, the disclosure will focuson FIGS. 18 through 20. These figures demonstrate a hybrid approach forperforming movement detection in conjunction with depth detection. Atthe end, the disclosure will turn to FIG. 21, which presents an examplecomputer system that may be used to facilitate the disclosed principles.

Improved Methodologies for Determining Depth

Attention is now directed to FIG. 1, which illustrates an exampleenvironment 100 of a user 105 using a HMD 110. The HMD 110 is an exampleof a mixed-reality system that is able to render virtual content for theuser 105. As previously noted, the HMD 110 may be a VR system or an ARsystem, such that environment 100 may be a VR environment or an ARenvironment. The term environment, mixed-reality environment andsurrounding environment will be used interchangeably herein to refer toenvironment 100 and other HMD environments referenced herein.

In world-locked holograms/mixed-reality environments (akaworld-stabilized imaging), a user may experience discomfort when his/herhead movement is not matched to what is visually displayed. Therefore,it is desirable to provide the user 105 with as pleasant an experienceas possible while the user 105 is wearing the HMD 110 by determining theuser's position in relation to the various objects in the environment100 (i.e. to perform depth detection and head tracking).

In FIG. 1, the environment 100 is shown as including a first object 115and a second object 120. To obtain an accurate mapping of the realobjects in the scene (aka mixed-reality environment), it is beneficialto know how far away these objects are from the user 105 at any givenmoment. By following the principles disclosed herein, significantadvantages are realized because highly accurate depth determinations maybe performed. By performing these depth determinations, themixed-reality environment, which is created by the HMD 110, canaccurately place virtual objects that interact with the real world. Thisresults in a more life-like interaction of virtual and real worldobjects, and the user 105's experience will be significantly improved.

FIG. 2 shows a HMD 200 that is specially configured to perform advanceddepth determinations in addition to rendering mixed-realityenvironments. For reference, this HMD 200 is one example implementationof the HMD 110 from FIG. 1. FIG. 2 is illustrated from a topperspective, looking down at the HMD 200, as indicated by the “x, y, z”direction legend.

As shown, HMD 200 includes a head-tracking stereo camera pair whichincludes at least two cameras, namely camera 205 and camera 210, both ofwhich are mounted on the HMD 200. According to the disclosedembodiments, the head tracking stereo camera pair may be used formultiple different operations, including, but not limited to, capturingimages for tracking the movements of the HMD 200, as well as capturingimages for determining depth.

Although HMD 200 is shown as including only two cameras, the HMD 200 mayactually include any number of cameras. For instance, the HMD 200 mayinclude 3 cameras, 4 cameras or more than four cameras. As such, the HMD200 is not limited only to two cameras.

Camera 205 is shown as including an optical axis 215. For reference, acamera's optical axis is an imaginary “line” that passes through thedirect center of the camera's lens. As a practical example, an opticalaxis is akin to the point where the camera is being aimed. In additionto the optical axis 215, FIG. 2 also shows that camera 205 has a fieldof view 220. In some implementations, camera 205 includes a wide-anglelens such that the field of view 220 is also a wide-angle field of view.This wide-angle field of view may span a range anywhere from 45 degreesup to 180 degrees horizontally (in ultra-wide-angle cameras) andanywhere from 45 degrees up to 120 degrees vertically.

Camera 210 may be configured similarly to camera 205. For instance,camera 210 similarly includes an optical axis 225 and a field of view230. By combining the fields of view of the two cameras, a very largespanning area (e.g., 170 degrees, 180 degrees, etc.) around the HMD maybe captured.

These cameras may be configured in many different ways. For example, insome implementations, both of the cameras 205 and 210 are configured asglobal shutter cameras. In other implementations, however, the cameras205 and 210 are configured as rolling shutter cameras. Of course,combinations of global shutter and rolling shutter cameras may also beused. As an example, the camera 205 may be a global shutter camera whilethe camera 210 may be a rolling shutter camera. In a preferredembodiment, a global shutter camera is used because rolling shuttercameras are more prone to motion blur. Of course, the HMD 200 may havemany cameras, some of which are global shutter and some of which arerolling shutter.

In some implementations, the cameras 205 and 210 (and in particular thepixels of these cameras) may be configured to detect, or rather besensitive to, different spectrums of light (e.g., visible light andinfrared (IR) light). For reference, the visible light spectrum rangesanywhere from around 380 nanometers (nm) up to and including about 740nm. More specifically, violet light ranges from 380 nm to 435 nm. Bluelight ranges from 435 nm to 500 nm. Green light ranges from 500 nm to520 nm. Yellow light ranges from 565 nm to 590 nm. Red light ranges from625 nm to 740 nm.

In contrast to visible light, infrared (IR) light is invisible to ahuman's eye and has a wavelength that is longer than the wavelengths forvisible light. The infrared light spectrum starts at the trailing edgeof the red light spectrum, around 700 nm, and extends to at least 1 umin length.

With that said, cameras 205 and 210 (at a pixel level) are configured todetect both visible light and IR light. In some instances, one or moreof the cameras 205 and 210 are monochromatic cameras (i.e. greyscale).In some instances one or more of the cameras 205 and 210 are chromaticcameras.

Of course, the cameras 205 and 210 may also be configured to detect onlyportions of the visible light spectrum and portions of the IR lightspectrum. This may be achieved through the use of one or more opticalbandpass filters in the lens. For brevity, the remaining disclosure willsimply use the singular form of the term bandpass filter even thougheach camera may be configured with its own similarly configured oruniquely different bandpass filter.

The bandpass filter is configured, in some instances, to allow only aselected range of visible light to pass through and be detected by oneor more corresponding camera(s) and while also allowing some or all IRlight to also be detected by the same camera(s). Additionally, oralternatively, the bandpass filter may be configured to allow only aselected range of IR light to pass through and be detected by the one ormore corresponding camera(s) while allowing some or all visible light topass through and be detected by the same camera(s).

By way of example, the bandpass filter is configured in some embodimentsto pass visible light having wavelengths between approximately 400 nm upto approximately 700 nm. In some embodiments, the bandpass filter isalso specifically configured to pass IR light having wavelengthscorresponding to the same wavelengths of IR light emitted by an IR lasermounted on the HMD 200 (to be discussed in more detail later). Oneexample of the IR laser's wavelength may be approximately 850 nm. Assuch, the bandpass filter may pass IR light having wavelengths within athreshold value of the IR laser's wavelengths (e.g., within 10 nm, 20nm, 30 nm, 40 nm, 50 nm, etc. of the emitted IR wavelength) while notpassing other IR light wavelengths.

In view of the foregoing, it will be appreciated that one or bothcameras 205 and 210 may include a bandpass filter that allows at leastsome visible light to pass through the bandpass filter (whilepotentially filtering out some visible light) and at least some IR lightto pass through the bandpass filter (while potentially filtering outsome IR light). Likewise, in some implementations, camera 205 and/orcamera 210 may also omit any IR light filter.

FIG. 2 also shows how the cameras 205 and 210 may be positioned inrelation to each other on the HMD 200. For example, at least a part ofthe field of view 220 of camera 205 is shown as overlapping at least apart of the field of view 230 of camera 210 thus forming the overlappingregion 235 (aka an “overlapping field of view region”). This overlappingregion 235 is beneficial for a number of reasons, which will bediscussed later.

In some configurations, the cameras may be horizontally offset (e.g.,offset relative to a horizontal alignment of the HMD 200 in they-direction plane). For instance, camera 205 may be pointed slightlydownward or upward in the y-direction while camera 210 may be alignedwith the horizontal plane (e.g., y-direction). In this manner, thecamera 205 may have a y-angle offset in relation to the horizontalalignment of the HMD 200. Relatedly, the camera 210 may be pointedslightly downward or upward in the y-direction relative to camera 205,while camera 205 is aligned with the y-direction horizontal plane. Ofcourse, combinations of the above are also available. For instance,camera 205 may be pointed slightly downward relative to the horizontalplane and camera 210 may be pointed slightly upward relative to thehorizontal plane, and vice versa. Alternatively, cameras 205 and 210 arehorizontally aligned, such that they do not have any y-angle offset andsuch that they are pointed directionally level in the y-direction.

Additionally, or alternatively, to the above horizontalalignments/offsets, cameras 205 and 210 may also be aligned/offset inother directions. For instance, FIG. 2 shows that the optical axis 215of camera 205 is angled (i.e. non-parallel) in relation to the opticalaxis 225 of camera 210 in the x-direction. Such a configuration issometimes beneficial because it allows the cameras 205 and 210 tocapture a larger area of the surrounding environment, thus providingmore reference area when performing movement detection (e.g., headtracking). This angle offset may be any selected angle. Example anglesinclude, but are not limited to 5 degrees, 10, 15, 20, 25, 30, 35, 40,45, 50, 55, 60, 65, 70, 75, 80, 85 degrees, and so on.

Although FIG. 2 and the remaining figures show the cameras 205 and 210angled in relation to one another, the embodiments should not be limitedto such a configuration. In fact, in some instances, the optical axes215 and 225 are aligned in parallel with one another in the x direction.In any event, and regardless of which orientation is used, the disclosedembodiments advantageously create overlapping region 235 with the fieldsof view 220 and 230.

Yet another configuration is available for the cameras 205 and 210. Toillustrate, the vertical positions of the cameras 205 and 210 (i.e. therelative height of the cameras along the y-direction on the HMD 200) mayalso vary. As an example, camera 205 may be positioned below camera 210on the HMD 200. Alternatively, camera 210 may be positioned below camera205 on the HMD 200. Otherwise, the cameras 205 and 210 are mounted atthe same relative height/vertical position on the HMD 200. Accordingly,from this disclosure, it is clear that the positions and orientations ofthe cameras 205 and 210 may vary widely.

Now that the configurations for the cameras 205 and 210 have beenintroduced, the disclosure will turn to how these cameras 205 and 210may operate. Recall, the stereo camera system/pair (i.e. cameras 205 and210) are configured to detect light for performing movement detection(e.g., head tracking, hand tracking, object tracking, etc.), as well asdepth detection. With regard to head tracking, the stereo camera pairactually constitutes an “inside-out” head tracking system because thestereo camera pair is mounted on the HMD 200.

An “inside-out” head tracking system tracks the position of a HMD (e.g.,HMD 200) by monitoring the HMD's position in relation to its surroundingenvironment. This is accomplished through the use of tracking cameras(e.g., cameras 205 and 210) that are mounted on the HMD itself and thatare pointed away from the HMD. In contrast, an “outside-in” trackingsystem uses cameras or external light illuminators that are mounted inthe environment and that are pointed toward the HMD. In this manner,inside-out head tracking systems are distinguished from outside-in headtracking systems.

As shown, cameras 205 and 210 are mounted on the HMD 200 (i.e. theobject being tracked) and may be (but are not required to be) slightlyoriented away from each other (as shown by the angled orientation of theoptical axes 215 and 225). Stated differently, the optical axis 215 isangled in relation to the optical axis 225.

To capture as much of the surrounding environment as possible, camera205 and camera 210 may be positioned apart, at a preselected distancefrom each other, and may be angled away from each other. Thispreselected distance is referred to as a “baseline,” and it may be anydistance. Commonly, however, the baseline will range anywhere between atleast 4 centimeters (cm) up to and including 16 cm (e.g., 4.0 cm, 4.1cm, 4.2 cm, 4.5 cm, 5.0 cm, 5.5 cm, 6.0 cm, 6.5 cm, 7.0 cm, 7.5 cm, 8.0cm, 8.5 cm, 9.0 cm, 9.5 cm, 10.0 cm, 10.5 cm, 11.0 cm, 11.5, cm, 12.0cm, 12.5 cm, 13.0 cm, 13.5 cm, 14.0 cm, 14.5 cm, 15.0 cm, 15.5 cm, 16.0cm, or more than 16.0 cm or less than 4.0 cm.). Often, the baseline isat least 10 centimeters. Sometimes, the baseline is chosen to match themost common interpupil distance for humans, which is typically between5.8 cm and 7.2 cm. In general, a wider baseline allows for accuratedepth from stereo for an increased distance over narrower baselinedesigns. Other factors that may influence the accuracy of the camerasystem are the cameras' fields of view and their image resolution.

With the foregoing configuration, the stereo camera system is enabled tocapture a large area of the surrounding environment, thus enabling theHMD 200 to interpolate its own position in relation to that environment.In addition to performing head tracking, the HMD 200 (and specificallythe stereo camera pair along with the stereo camera pair's logicalcomponents) may be re-purposed, or rather multi-purposed, to alsoperform an improved form of depth detection. By re-purposing existinghardware components, the embodiments significantly reduce the cost forperforming depth detection, especially when compared to time-of-flightdepth detection systems.

As an initial matter, it is noted that humans are able to perceive“depth” because humans have a pair of eyes that work in tandem. Whenboth eyes are focused on an object, signals from the eyes aretransmitted to the brain. The brain is then able to interpolate depthusing any disparity existing between the information captured from thetwo eyes.

Similar to how a human's eyes “focus” on an object when determiningdepth, the HMD 200 also obtains “focused” digital image content todetermine depth. Here, the “focused” digital image content is obtainedfrom camera images that include content corresponding to the overlappingregion 235 (i.e. camera 205's image and camera 210's image, both ofwhich include digital content corresponding to the overlapping region235). In this manner, the cameras 205 and 210 obtain separate images,but these images still have at least some similar content.

Here, an example will be helpful. Suppose a table was located in the HMD200's environment and that the HMD 200 was positioned so that the tablewas located within the overlapping region 235. In this scenario, cameras205 and 210 are each able to obtain a digital image that includesdigital content corresponding to the table. Consequently, at least someof the pixels in the image obtained by camera 205 will correspond to atleast some of the pixels in the image obtained by camera 210.Specifically, these “corresponding pixels” (i.e. the pixels in the oneimage that correspond to the pixels in the other image) are associatedwith the table.

Once these digital images are obtained, then the HMD 200 performscertain transformations (also called “re-projections”) on those digitalimages. These transformations correct for lens distortion and othercamera artifacts. Furthermore, the stereo images are re-projected onto avirtual stereo rig where both image planes lie inside a plane that isparallel to the stereo cameras' baseline. After re-projection,corresponding pixels are guaranteed to lie on the same horizontalscanline in left and right images. As a result, two “re-projected”images are formed, one for the image that was obtained by the camera 205and one for the image that was obtained by the camera 210. Any pixelsthat are similar/correspond between the two re-projected images now lieon the same horizontal plain.

After the re-projected images are created, the HMD 200 measures anypixel disparity that exists between each of the corresponding pixels inthe two images. Because the HMD 200 understands that the correspondingpixels in the two re-projected images are now in the same horizontalplain, the HMD 200 identifies that the disparity between thesecorresponding pixels corresponds (i.e. is proportional) with a depthmeasurement. Using this disparity, the HMD 200 assigns a depth value toeach pixel, thus generating a depth map for any objects located in theoverlapping region 235. Accordingly, the HMD 200, through the use of itsmulti-purposed head-tracking stereo camera pair, is able to perform bothmovement detection as well as depth detection.

The remaining portion of this disclosure uses many examples of camerasand head tracking stereo camera pairs (or simply stereo camera pairs).Unless stated otherwise, these cameras may be configured with any of thepositional/alignment configurations discussed above. Therefore,regardless of whether the system is performing head tracking or depthdetection, any of the cameras mentioned above, operating in any of theconfigurations mentioned above, may be used.

With that understanding, attention will now be directed to FIG. 3. Inthis illustration, an example environment 300 is provided, which may bepresented to a user (e.g., user 105 from FIG. 1) who is using a HMD(e.g., HMD 110 from FIG. 1 or HMD 200 from FIG. 2) to visualize amixed-reality environment.

Environment 300 includes a number of different features and objects. Forexample, environment 300 includes a textureless/smooth table top 305, atextureless/smooth wall 310, and a textured door frame 315, just to namea few. Of course, this is just one example of what an environment maylook like, and thus should not be considered limiting or otherwisebinding.

One problem that conventional depth perception systems have faced isdetermining depth for “textureless/smooth” objects (e.g., thetextureless/smooth table top 305 and the textureless/smooth wall 310).For textured surfaces, like the textured door frame 315, traditionaldepth detection systems are usually able to capture enough details toperform the stereo matching between the left and right cameras toadequately gauge the depth of those textured objects. Unfortunately,however, traditional depth detection systems are very inadequate indetermining the depth of textureless/smooth objects. In particular,traditional depth detection systems cannot collect enough information toadequately distinguish one part of the textureless/smooth object fromanother part, which may be further away.

For instance, if a user were to stand near the textureless/smooth tabletop 305, portions of the textureless/smooth wall 310 will besignificantly closer than other portions of the textureless/smooth wall310. However, traditional systems are unable to account for this changein depth because of a lack of texture on the surfaces and, hence a lackof reflected light that is used to determine the depth. As a result,traditional systems will often generate a false or otherwise misleadingdepth map for textureless/smooth objects like textureless/smooth wall310. If any virtual content is dependent on that false depth map, thenclearly the mixed-reality environment will be skewed and thus the user'sexperience will be hampered.

To address the above problems, some of the disclosed embodimentsbeneficially project, or rather add, texture to the environment. In someimplementations, this texture is in the form of an infrared (IR)dot-pattern illumination. Because the HMD's stereo camera pair (e.g.,camera 205 and 210 from FIG. 2) is sensitive to both visible andinfrared (IR) light, the stereo camera pair is able to detect the addedtexture and compute proper depth for any kind of object, even atextureless/smooth object. The HMD is thereby provided a picture withstructured light, thus improving depth quality determinations.

Attention is now directed to FIG. 4. In this illustration, an IRdot-pattern illuminator 400 projects/disperses IR light as an IRdot-pattern illumination 405. This IR dot-pattern illumination 405 maybe projected to any predetermined illumination area within the HMDsurrounding environment (e.g., any area in the environment 300 from FIG.3).

Although, FIG. 4 only shows a single IR dot-pattern illuminator 400being used to project the IR dot-pattern illumination 405, it will beappreciated that the IR dot-pattern illuminator may actually comprisetwo or more IR dot-pattern illuminators, (not shown), and which aremounted on the HMD. It will also be appreciated that the IR dot-patternilluminator 400 may also include any combination of IR light emittingdiode (LED), LED array, IR laser diode, incandescent dischargeilluminator, vertical-cavity surface-emitting laser (VCSEL) and/orplasma discharge illuminator.

The IR dot-pattern illumination 405 may be generated in various ways.For instance, in a preferred embodiment, the IR dot-pattern illumination405 is generated using a diffraction limited laser beam, a collimatingoptic, and a diffractive optical element (DOE). As such, the IRdot-pattern illuminator 400 may also include a collimating optic and aDOE to provide the desired projection/dispersion of the IR dot-patternillumination 405. When an IR laser shoots a diffraction limited laserbeam of IR light into the DOE, then the DOE disperses the IR light insuch a manner so as to project the pre-configured dot patternillumination. Other IR LED, incandescent discharge illuminator, VCSEL,plasma discharge illuminator, etc., may be used with more traditionalimaging and re-projection techniques as well.

In an alternative embodiment, an etched lens may also be placed over topof an IR optical source/illuminator. In a first example, individual dotsmay be etched onto the lens to create the dot pattern. When thedot-pattern illuminator 400's IR laser emits a beam of IR light throughthis type of lens, the IR light unimpededly passes through the lens inthe areas that were not etched. However, for the dot areas that wereetched, the IR light may be impeded in accordance with the etchedpattern, thus projecting a dot pattern into the surrounding environment.

In a second example, large swatches may be etched onto the lens whileavoiding small “dot” areas that correspond to the dot pattern. When theIR laser emits a beam of IR light through this type of lens, only IRlight that passes through the small unetched “dot” areas will passunimpededly, thus projecting a dot pattern into the surroundingenvironment. Any other technique for generating a dot pattern may alsobe used (e.g., instead of etching the lens, a dot-pattern covering maybe placed on the lens). Additionally, any other DOE may be used todisperse IR light in accordance with a pre-configured dot pattern.Regardless of its implementation, a beam of IR light is dispersedaccording to a predetermined dot-pattern.

While the disclosure has used the phrase “dot pattern,” it will beappreciated that the term “dot” does not limit the illuminations to acircular shape. In fact, any shape of dot may be projected in thepredetermined dot-pattern. For example, the dot pattern may include apattern of circles, triangles, squares, rectangles and/or any otherpolygon or oval shaped dot(s).

It will also be appreciated that any kind of dot “pattern” may be used.For instance, the pattern may be a completely random assortment of dots.Alternatively, the pattern may include a pre-configured pattern thatrepeats itself in a horizontal and/or vertical direction. By repeatingthe dot pattern at least in the vertical direction, advantages arerealized because the accuracy of the stereo matching algorithm will beimproved. Even further, the size of the dots may also vary such thatsome dots may be larger or smaller than other dots to facilitate patternmatching.

As described herein, the IR dot-pattern illumination 405 is projectedinto the surrounding environment of a HMD in order to project, or ratheradd, “texture” to object surfaces in the surrounding environment. Forinstance, FIG. 5A shows an example environment 500 in which IRdot-pattern illuminator 505 is projecting an IR dot-pattern illumination510 onto at least a part of the environment 500. As such, this IRdot-pattern illumination 510 is adding artificial texture to the objectsin the environment 500. FIG. 5B shows how the cameras may be oriented inrelation to the IR dot-pattern illuminator and how the cameras' fieldsof view overlap, which will be discussed in more detail in connectionwith FIG. 8.

FIGS. 6A and 6B show another example usage. In particular, FIG. 6A showsan environment as viewed according to the visible light spectrum 600A.In this environment, there is a textureless/smooth table top 605A and ahand 610A that is partially occluding a portion of thetextureless/smooth table top 605A. Conventional depth detection systemsmay have a difficult time in segmenting (i.e. distinguishing) thetextureless/smooth table top 605A from the hand 610A because thetextureless/smooth table top 605A does not have a sufficient amount oftexture for a convention system to determine depth.

Turning now to FIG. 6B, there is the same environment, but shown in acombination of visible and IR light spectrum 600B. In particular, theenvironment includes a textureless/smooth table top 605B and a hand610B. In this embodiment, IR dot-pattern illuminator 615 projects an IRdot-pattern illumination 620 onto both the textureless/smooth table top605B and the hand 610B. As discussed in more detail below, this IRdot-pattern illumination 620 provides additional texture so that thestereo camera system of the HMD can more accurately generate a depth mapfor that environment.

Attention is now directed to FIG. 7. As shown, an IR dot-patternilluminator 700 is mounted on a HMD. Because the elements of the HMD inFIG. 7 are very similar to the elements shown in FIG. 2 (e.g., the HMD200, the cameras 205 and 210, and the fields of view 220 and 230), thecommon elements have not been relabeled.

Here, the IR dot-pattern illuminator 700 is oriented in such a manner asto emit the IR dot-pattern illumination 705 to at least partiallyoverlap with the overlapping field of view region 710 of the HMD camerasystem. In some implementations, the IR dot-pattern illumination 705overlaps a majority and/or all of the overlapping field of view region710, while in other implementations, the IR dot-pattern illumination 705overlaps only a minority portion of or other selected percentage (e.g.,30%, 35%, 40%, 45%, 50%, 55%, 60%, 70%, 80%, 90%, etc.) of theoverlapping field of view region 710.

The overlap over the IR dot-pattern illumination 705 with the field ofview region 710 enables both of the HMD's cameras to detect at least apart of the IR dot-pattern illumination 705 being reflected off ofobjects in the overlapping field of view region 710. In this manner, thecameras are able to obtain digital images that include digital contentcorresponding to the texture (i.e. the “obtained” texture is actuallyreflected IR light generated as a result of the IR dot-patternillumination 705 reflecting off of surfaces in the environment). Usingthe left and right camera images with improved details, the stereomatching is improved, allowing the depth detection system to computepixel disparity, thus determining depth of objects (eventextureless/smooth objects) in the overlapping field of view region 710.Such functionality is shown and described in more detail below.

As shown in FIG. 8, a hand 800 and a table 805 corresponding to a depthmap are illustrated. This depth map was generated by determining thepixel disparity of corresponding images taken by the camera stereosystem of the HMD and then mapping the resulting depths to eachcorresponding pixel, as discussed previously. In this exampleillustration, because a majority of the hand 800 is substantially all inthe same dimensional plain, the depth determinations for most of thehand 800 will be the same. As such, the hand 800 is illustrated with arelatively consistent depth shadowing.

In contrast, because the table 805 is not all in the same dimensionalplane, it will have differing depths. These differing depths areillustrated in FIG. 8 with correspondingly different shadowing gradientsassociated with the different depth patterns of the table. Inparticular, the closer an object is to the HMD, the darker the objectis. Thus, because the hand 800 is much closer to the HMD, the hand 800is shown in a very dark color. Similarly, the portions of the table 805that are closest to the HMD are also in a dark color. As the distancebetween the HMD and the table 805 increases, the color gradientprogressively goes from dark to light. Accordingly, FIG. 8 provides anillustrative example of the differences in depth that objects may haveand how those depths may be determined by the HMD.

As previously noted, it can sometimes be difficult to detect and map thedepths of textureless/smooth objects. However, utilizing the techniquesdescribed herein, it is possible to supplement the texturing of theobjects being mapped with IR dot-pattern illuminations to therebyenhance/improve the depth mapping that is performed.

Of course, it will be appreciated that the illustration shown in FIG. 8is simply being used as an example to particularly emphasize the“visualization” of depth. To clarify, these depth shadowing gradientsare not actually created by the HMD. Instead, the HMD uses a depth mapthat maps each pixel to one or more dimensional values. In this manner,the HMD is able to identify and record an object's varying degrees ofdepth. This depth map records not only horizontal and verticaldimensions for each pixel in an image, but it also records a depthdimension for each pixel in the image. Stated differently, and usingconventional coordinate naming techniques, the depth map records an “x”coordinate value, a “y” coordinate value, and a “z” coordinate value foreach pixel. Therefore, the HMD uses a depth map as opposed to thegradient scheme shown in FIG. 8.

As described, the stereo camera images (e.g., images that includecontent corresponding to the reflected IR dot-pattern light and visiblelight) are used to generate a depth map that is calculated via stereotriangulation (e.g., by at least determining disparity between the twoimages). In many embodiments, the depth map is generated for at least apart of a mixed-reality environment generated by the HMD. For example,the mixed-reality environment may include the user's hand, a table, orany other object. As such, the depth map may include a determinationregarding a size and a pose of the user's hand, the table, or the otherobjects.

The foregoing texturing/depth mapping techniques described herein canalso be beneficial for helping to perform segmentation of objects thatare occluded in the field of view and to further facilitate tracking ofthe objects. In particular, the HMD may utilize the supplementedtexturing (i.e., IR dot-pattern illuminations) to generate enhanceddepth maps. In particular, the enhanced depth maps can includeadditional depth data that is based on reflected IR light and thatsupplements the data that is based on only visible light. This can beparticularly useful in low ambient light conditions. In this manner, itis possible to further facilitate segmenting objects in a digital image,such as the illustrated user's hand 800 from the illustrated table 805.

The term “segmented” means that the two objects are distinguished fromone another even though the hand is overlapping, or rather occluding, aportion of the table, relative to the perspective view of the HMD. Thesegmenting is performed by computing the depths for some or all of thepixels in the digital images that are obtained by the HMD camera system,such as by detecting the reflected visible light and the reflected IRdot-pattern illuminations, and to thereby distinguish one object fromanother based on their relative depths.

In some instances, segmenting can also be performed by analyzing digitalimages obtained from the reflected IR dot-pattern illuminations todetected known object shapes (e.g., a hand) through machine learning.For instance, the HMD system can reference object shape definitionsstored locally or remotely, which match detected shapes in the digitalimages.

In some implementations, the depth map is also dynamically adjusted toreduce or account for a potential “jitter” phenomenon. Depth jitter mayoccur when neighboring pixels have vastly different depth coordinatevalues. Sometimes, these large differences cause the renderedvisualizations to appear strange because the HMD may try to repeatedlyadjust those visualizations to account for the large depth disparities.Such adjustments may cause the visualizations to “jitter” between aclose position and a far position. Stated differently, these differencesin determined depth may cause blatantly obvious distortions in thequality of any visualizations provided to the user.

To reduce the likelihood of a jitter, some embodiments perform filteringon the depth map. Specifically, these embodiments filter out neighboringcoordinate pixels that do not share sufficiently similar depth values.“Similar,” in this connotation, means that the coordinate values of theneighboring pixels are within a particular threshold value of eachother. Of course, when those depth coordinates are filtered out of thedepth map, they may be replaced with new depth coordinates. These newdepth coordinates are selected to satisfy the threshold requirement. Byperforming these filtering operations, the amount of flickering that mayoccur will be significantly reduced, and thus the user will have a muchmore pleasant experience.

Attention is now directed to FIG. 9, which is provided to illustrate anddescribe how the IR dot-pattern illuminator 905 can be mounted to theHMD 930 with various configurations relative to the stereo cameras 900and 910.

Up to this point, the figures have shown that the IR dot-patternilluminator is positioned roughly an equal distance between both camerasin the stereo camera pair. Other configurations are also available,however, as shown in FIG. 9. In this illustration, left camera 900, theIR dot-pattern illuminator 905, the right camera 910 are mounted on aHMD, with the circle labeled “A” illustrating one example position ofthe left camera 900 on the HMD, the circle labeled “C” illustrating oneexample position of the right camera 910 and the squares labeled “B”illustrating possible positions of the IR dot-pattern illuminator 905.Notably, the IR dot-pattern illuminator 905 may be positioned anywherein between or outside of the left camera 900 and/or the right camera910.

To further clarify, in some implementations, the IR dot-patternilluminator 905 is between the left camera 900 and the right camera 910.In other implementations, the IR dot-pattern illuminator 905 ispositioned closer to the left camera 900 than to the right camera 910.In yet other implementations, the IR dot-pattern illuminator 905 ispositioned closer to the right camera 910 than to the left camera 900.Furthermore, the ellipses 915 and 920 demonstrate that the IRdot-pattern illuminator 905 may also be positioned further outside theperiphery of either the left camera 900 or the right camera 910.

It will also be appreciated that the IR dot-pattern illuminator 905 maybe a distributed illuminator that is simultaneously present/mounted atdifferent B positions on the HMD, relative to the cameras 900 and 910.

FIG. 9 shows that one baseline 940 exists between the left camera 900and the right camera 910. In addition to this baseline 940, there isalso a baseline 950 between the left camera 900 and the IR dot-patternilluminator 905, as well as a third baseline 960 between the IRdot-pattern illuminator 905 and the right camera 910. By placing the IRdot-pattern illuminator 905 at any of the positions marked as “B,” theseother baselines change. In some implementations, the IR dot-patternilluminator 905 may beneficially act as a “virtual camera” having itsown baseline(s). Further detail is provided below.

To illustrate, when the IR dot-pattern illuminator 905 is close to theleft camera 900, then the baseline 950 between the IR dot-patternilluminator 905 and the left camera 900 is smaller than the baseline 960between the IR dot-pattern illuminator 905 and the right camera 910. Insome circumstances, this is beneficial to have unequal “virtual camera”baselines (i.e. the IR dot-pattern illuminator 905's baselines) becauseunequal baselines may actually improve the detection of the texture(e.g., by providing different disparity calculations and by providing amechanism for generating a virtual camera image based on the twoadditional baselines 950 and 960) and to thereby help to improve theoverall depth determinations.

As an example, it is possible to store an image of the dot pattern asobserved from the perspective of the IR dot-pattern illuminator 905 whenthe IR dot-pattern illuminator 905 is positioned at any of the “B”locations, based on disparities/baselines between the IR dot-patternilluminator 905 and the corresponding cameras. This image will beconstant (i.e. not dependent on environmental content) and will yield athird camera that is purely virtual. The HMD can then match the virtualpre-recorded image with real camera images. In comparison to thetwo-view stereo camera algorithm, the HMD (when using the virtual cameraimage) can produce depth results that are higher in quality. When usingsuch a setup, it is beneficial to mount the IR dot-pattern illuminator905 at an off-center position (i.e. any of the positions marked “B”except for the central-most position). By doing so, the HMD is able toobtain the three different baselines mentioned earlier. Notably, largerbaselines are preferred for reconstructing large depths whereas smallbaselines are preferred for reconstructing small depths.

While FIG. 9 illustrates the variable positioning of the components froma top aerial view, it will be appreciated that the position of IRdot-pattern illuminator 905 may also vary vertically (i.e. in they-direction) on the HMD. For instance, the IR dot-pattern illuminator905 may be mounted higher or lower than the left camera 900 and/or theright camera 910 on the HMD. Alternatively, the IR dot-pattern 905 maybe mounted levelly in the y-direction with both the left camera 900 andthe right camera 910.

Attention is now directed to FIGS. 10 through 12, which illustratevarious example computer system components and various different methodsthat may be performed to improve depth detection with stereo trackingcameras. In FIG. 10, various depth detector computer system components1000 are illustrated, which may be integrated into a HMD. For instance,these components may be part of the HMD 200 that was discussed inconnection with FIG. 2 or any of the other HMDs discussed thus far.

Here, the depth detector computer system components 1000 include cameras1005 which may be an example implementation of the left camera 900 andthe right camera 910 from FIG. 9 or any of the other cameras discussedherein. Additionally, the depth detector computer system components 1000include an IR dot-pattern illuminator 1010 which may be an exampleimplementation of the IR dot-pattern illuminator 905 from FIG. 9 or anyof the other IR dot-pattern illuminators discussed herein.

The various depth detector computer system components 1000 also includea head tracker 1015 and a depth detector 1020, as well as one or moreprocessors 1025 and storage 1030. The one or more processors 1025 mayinclude hardware processors, each of which may be specially configuredprocessor (e.g., a graphics processing unit (GPU) or as an applicationspecific integrated circuit (“ASIC”)).

In some instances, the processor(s) 1025 include one processor that isconfigured to operate as the head tracker 1015 and an additionalprocessor that is configured to operate as the depth detector 1020.Alternatively, a single processor is used to operate as both the headtracker 1015 and the depth detector 1020. More details on processors andhow they operate will be provided in connection with FIG. 21.Additionally, the head tracker 1015 and the depth detector 1020 will bedescribed in more detail in conjunction with the methods illustrated inFIGS. 11 and 12. The processor(s) 1025 also execute storedcomputer-executable instructions stored in the storage 1030 forimplementing the acts described herein.

The following discussion now refers to a number of methods and methodacts that may be performed. Although the method acts may be discussed ina certain order or illustrated in a flow chart as occurring in aparticular order, no particular ordering is required unless specificallystated, or required because an act is dependent on another act beingcompleted prior to the act being performed.

As shown in FIG. 11, a flowchart 1100 is provided with various actsassociated with methods for adding predetermined texture to objectslocated within an environment of a HMD. The first illustrated act is anact for emitting an IR dot-pattern illumination from an IR dot-patternilluminator (act 1105), such as an illuminator that is mounted on a HMD.For reference, this act may be performed by the IR dot-patternilluminator 1010 from FIG. 10.

In some implementations, the IR dot-pattern illuminator pulses the IRdot-pattern illumination at a high peak power. By pulsing at the highpeak power, the pulsed IR light is able to compensate for ambient lightnoise in both the visible light spectrum and the IR light spectrum. Toclarify, the IR dot-pattern illuminator's IR laser operates, in someembodiments, in a pulsed mode. When operating in a pulsed mode, the IRlaser will repeatedly cycle on and off, or rather emit IR light and thennot emit IR light, to thus create a pulsing emission of IR light. Insome embodiments, the IR dot-pattern illuminator pulses the IRdot-pattern illumination for an exposure time between 0.5 and 5milli-seconds, and this illumination pulse is synchronized with thecamera exposure time (as described below). The repetition rate of thepulsed laser corresponds to the frame rate of the camera module, whichis typically between or including 0.5 frames per second and 10 framesper second. Pulsing is beneficial because it reduces the amount of timethat the laser is operating, thus reducing battery consumption.Furthermore, pulsing the IR light at the high peak power allows the IRlight signal to be more detectable and to thereby overcome any ambientlight noise. The use of bandpass filters can also help to filter outambient light noise.

On a related note, when the IR laser is operating in a pulsed mode, thenthe stereo camera pair is synchronized with the IR laser's pulsing. Forexample, the cameras will obtain images in sync with the IR laseremissions, timed to detect the corresponding IR light reflections. Inmore particular, the pulsing of the IR dot-pattern illumination issynchronized, in some embodiments, with the exposure time of thecameras. The exposure times of the cameras may also be increased ordecreased in order to increase the number of photons that are capturedin the cameras' images. This can be particularly beneficial whenperforming tracking in a low light environments, which will be discussedin more detail below.

The exposure times of the cameras may also vary depending on whichoperation, or rather mode, the HMD is operating in, be it a headtracking mode, a hand tracking mode, or a depth detection mode. For headtracking operations, example exposure times range anywhere between 0.016milli-seconds up to and including 25 milli-seconds. Of course, theexposure time may vary within this range such that the HMD may notalways use the same exposure time for all head tracking images. Forinstance, if the HMD determines that a sufficient/predetermined amountof photons have been sensed to generate an image that is usable forperforming tracking or depth detection, then the HMD willselectively/dynamically reduce its cameras' exposure time. Similarly, ifthe HMD determines that additional photons are needed to generate animage that is usable for performing tracking or depth detection, thenthe HMD will selectively/dynamically increase the exposure time. In thismanner, the HMD may use a variable exposure time. In some embodiments,the HMD will dynamically vary its cameras' exposure times depending onwhich operation, or rather mode, the HMD is operating in (e.g., trackingmode, depth detection mode, burst mode, pulsed mode, etc.).

For depth detection operations, example exposure times will be set, insome embodiments, to range anywhere from 0.5 milli-second up to andincluding 5 milli-seconds. Furthermore, it will be appreciated that insome embodiments, the exposure times for two or more of the cameras aresynchronized with each other such. Additionally, in some embodiments theillumination time, whether for the IR dot-pattern illuminator or for theflood IR illuminator, overlaps the entirely of the cameras' exposuretimes to maximize the amount of light captured by the cameras.

In some embodiments, the HMD initially uses its cameras to detect anamount of reflected IR light generated as a result of the IR dot-patternilluminator emitting the IR dot-pattern illumination. In response to adetected “intensity” of the reflected IR light and/or ambient light,which intensity may be detected using the cameras, the HMD modifies apower level of the IR dot-pattern illuminator until a predeterminedthreshold intensity of the reflected IR light is detected by thecameras. As such, the HMD is able to dynamically adjust the power levelof the IR dot-pattern illuminator in order to increase or decrease theintensity of the IR light, when necessary (e.g., in low lightconditions) to obtain a desired level of detail in the captured imagesused for depth detection and/or tracking. Such adjustments areparticularly beneficial for enabling the HMD to use only as much poweras necessary in obtaining sufficiently adequate images for determiningdepth. Further, adjusting the illumination intensity may be performeddifferently for head tracking versus hand tracking, because hands aretypically located within 1 meter of the HMD, the HMD can emit IR lightwith a lower intensity when performing hand tracking (where the user'shand is typically within 1 meter of the HMD), as compared to the higherintensity of IR light that is emitted when performing head tracking andit is necessary to flood the environment with more light to gatherinformation from objects that are more than 3 meters away from the HMD.

Returning to FIG. 11, reflected IR light is subsequently detected (act1110). In addition to detecting reflected IR light, the cameras alsodetect visible light. This act is performed by the cameras 1005described in FIG. 10 and throughout this paper. Notably, this reflectedIR light is generated as a result of at least a portion of the IRdot-pattern illumination reflecting off of the surface in theenvironment (e.g., the textureless/smooth table top 605A from FIG. 6A).

To detect this reflected IR light, a first image, which includes digitalcontent corresponding to a first part of the IR dot-patternillumination, is captured using one of the cameras. Concurrently withthat operation, a second image is also captured using the second camera.The second image includes digital content corresponding to a second partof the IR dot-pattern illumination and which includes digital contentcorresponding to at least some of the same IR dot-pattern illuminationthat was captured in the first image. Furthermore, the images alsoinclude reflected visible light.

The first image and the second image are then used to determine a depthfor at least the common part of the illumination area that was capturedin both images (act 1115). By obtaining images of the reflected IR lightusing both a left camera and a right camera, the HMD is able to measurethe pixel disparity present between common pixels in the two images.This pixel disparity may then be used by the depth detector 1020 of FIG.10 to determine the stereoscopic depth for that object.

Notably, HMD is able to determine a depth for objects in theenvironment, even when those objects have textureless/smooth surfacesbecause of the IR dot-pattern that adds texture to the objects. Bydetermining the depth for objects in this manner (i.e. stereo vision),it is not necessary to utilize additional hardware/components to performdepth via time-of-flight. For reference, time-of-flight systems are muchmore expensive because they use additional hardware components. Incontrast, the current embodiments re-purpose many existing components sothat they can perform new or additional functionalities, thus savingsignificant costs and thereby reducing computational and power burdenson the HMDs.

FIG. 12 illustrates another example flowchart 1200 of methods that maybe performed for performing geometric surface reconstruction inconjunction with head tracking. Similar to the methods described inreference to FIG. 11, the methods of FIG. 12 may be performed by thedepth detector computer system components 1000 shown in FIG. 10.

As illustrated, method 1200 is shown as being dispersed in two differentcolumns (e.g., the column of acts under the label “Frequency A” and thecolumn of acts under the label “Frequency B”). Such formatting wasselected to show that the various acts may not be temporally dependenton one another.

With that understanding, method 1200 is shown as including a number ofvarious different acts that may be performed at a first selectedfrequency (i.e. Frequency A). Initially, the HMD's head tracking stereocamera pair is used to obtain one or more visible light images of theHMD's surrounding environment (act 1205). This may be accomplished withthe cameras described herein. Using these images, the HMD is able toidentify a number of reference targets (e.g., anchor points) in themixed-reality environment. Using these reference targets, the HMD isthen able to track position and orientation, as described below.

After identifying a number of reference targets (i.e. anchor points)from within the images (to be discussed in more detail below), the oneor more visible light images are then used to track a position of theHMD within the surrounding environment (act 1210). This may beperformed, for example, with the head tracker 1015 of FIG. 10.

As further illustrated, the methods of FIG. 12 may also includeadditional acts that are performed at a second selected frequency (i.e.Frequency B). Notably, Frequency B may be different than Frequency A,such that the camera images are taken at different time stamps. Forexample, the acts under Frequency A correspond to head tracking and maybe performed at a rate of 30 frames per second (hand tracking isdiscussed later and may be performed at yet another frequency that ismore than 30 frames per second or a variable frequency). In contrast,the acts under Frequency B correspond to depth detection and may beperformed at a rate of less than 30 frames per second, such as 5 framesper second.

It will be appreciated that the referenced frequencies are illustrativeonly and non-limiting. For example, Frequency A may be 15, 20, 25, 30,35, 40, 45 frames per second, or other frame rates, and Frequency B maybe 0.5, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 frames per second, or other framerates. In some implementations, the second selected frequency (i.e.Frequency B) is lower than the first selected frequency (i.e. FrequencyA). As such, sensing the reflected IR light at the second selectedfrequency (i.e. the operations under the “Frequency B” label) may beinterleaved among the operations of obtaining the images of thesurrounding environment at the first selected frequency (i.e. theoperations under the “Frequency A” label).

Returning to FIG. 12, the HMD's IR dot-pattern illuminator emits an IRdot-pattern illumination onto at least a part of the surroundingenvironment (act 1215). This may be performed by the IR dot-patternilluminators described herein.

Next, the head tracking stereo camera pair senses “reflected” IR lightand/or reflected visible light (act 1220). This act may be performed bythe cameras described herein, which are configured to detect/sensevisible light and IR light.

Subsequently, a depth map is calculated/generated, using stereo depthimaging (act 1225). This depth map is based on the reflected IR lightand/or the reflected visible light that is captured by the stereo camerapair. This act may be performed, for example, by the depth detector 1020from FIG. 10.

Finally, a geometric surface may be constructed for the surroundingenvironment (specifically the part that was illuminated by the IRdot-pattern illumination) using the depth map (act 1230). This act maybe performed by the depth detector 1020 from FIG. 10.

It will be appreciated that the application and use of IR dot-patternillumination and repurposed use of head tracking cameras cansignificantly improve the quality of depth calculation for HMDs, andwhile reducing costs of the HMDs. Such processes are also particularlybeneficial when the environment surrounding the HMD includestextureless/smooth surfaced objects.

Improved Methodologies for Tracking Movement, Particularly in a LowLight Environment

Most mixed-reality HMDs are portable for use in different locations andenvironmental conditions. For example, the HMD may be worn in brightlylit environments, such as outside during broad daylight, as well as indark or low light conditions, such as at night or in a dark room.However, conventional systems have been quite deficient when it comes totracking movement in a low light environments.

As briefly discussed earlier, to perform tracking (i.e., movementdetection), the HMD often uses at least two cameras to record images ofthe surrounding environment. In particular, the HMD uses its cameras toidentify regions with high spatial information (e.g., areas that includea large amount of fixed features suitable to act as a reference), andthen the HMD uses this high spatial information to identify “anchorpoints.” To clarify, the cameras' images are analyzed to identify one ormore anchor points within that environment. An anchor point is anon-moving physical feature that can serve as a reference point indetermining the HMD's position in relation to the environment. As theposition of the HMD changes, the HMD is able to use information frominertial measurement units (IMU), gyroscopes (for high frequencychanges), accelerometers, and the cameras' images (for low frequencyanchoring) to detect the changing position and orientation of the HMDrelative to the anchor points.

An example of an anchor point includes, but is not limited to, a cornerof an object located in an environment surrounding the HMD. When the HMDobtains images and then analyzes those images, it can identify anchorpoints within that image. When a sufficient number of anchor points areidentified, then the HMD can determine its position and orientation inthe environment relative to the identified/mapped anchor points.Problems arise, however, when the environment is a low light environmentbecause the head tracking cameras may not have a sufficient number ofphotons to accurately generate a camera image of the scene, and to beable to adequately identify the anchor points within the images. Assuch, traditional movement detection solutions have been quite deficientwhen used in low light environments.

To address these needs, some of the disclosed embodiments make use of aflood IR illuminator, which is different than an IR dot-patternilluminator. Recall, the cameras in the stereo camera pair are able todetect both visible light as well as IR light. By projecting a flood ofinfrared light, the cameras are able to capture images of reflected IRlight and use those images to help with the process of performingmovement detection, including head tracking, hand tracking, etc. Suchuse is particularly beneficial when images that capture visible lightare inadequate to detect a sufficient number of anchor points (thistopic will be covered in more detail later). As such, the images thatcapture the flood of IR light may be used to augment and/or replace thevisible light images when identifying anchor points.

Attention will now be directed to FIGS. 13A through 17. These figuresillustrate various different example architectures, methods, andsupporting illustrations for improving the process of head tracking,particularly within low light environments.

Turning first to FIG. 13A, a low light environment 1300 is illustrated.For reference, a bright sunny day typically has an ambient lightintensity of around 10,000-50,000 lux. An overcast day typically has anambient light intensity of around 1,000-10,000 lux. An indoor officetypically has an ambient light intensity of around 100-300 lux. The timeof day corresponding to twilight typically has an ambient lightintensity of around 10 lux. Deep twilight has an ambient light intensityof around 1 lux. As used herein, the illustrated low light environment1300 least corresponds to any environment in which the ambient lightintensity is at or below about 40 lux. The HMD has one or more sensorsthat are configured to determine the surrounding environment luxintensity. These sensors can be incorporated into or independent fromthe cameras and/or illuminators described herein.

FIG. 13A also illustrates a flood IR illuminator 1305. As used herein,the flood IR illuminator 1305 includes one or more of an IR LED, anarray of IR LEDs, an IR laser diode, an incandescent dischargeilluminator, a VCSEL, a plasma discharge illuminator, etc. The flood IRilluminator 1305 also includes, in some embodiments, a correspondingcollimating optic. The IR illuminator is configured to emit abroad-beamed, homogeneous ray of light that spans a large illuminationarea. By “homogeneous,” it is meant that all illumination points aresubjected to substantially the same amount of light.

In this manner, the flood IR illuminator 1305 is configured to emit aflood of IR light spread across a selected field of view that at leastpartially illuminates the head tracking camera's FOV. This field of viewmay be spanned differently. For example, the field of view may span 90degrees, 100 degrees, 110 degrees, 120 degrees, 130 degrees, and so on.Additionally, the flood IR illuminator 1305 may be but one of an arrayof multiple illuminators.

As shown in FIG. 13A, the flood IR illuminator 1305 is emitting a floodof IR light 1310 into the environment 1300. Such a process isparticularly beneficial because it helps illuminate potential anchorpoints that may be used to track movements.

Such a scenario is shown in FIG. 13B. It will be appreciated that thecameras are able to detect both visible light and IR light. In someinstances, the cameras capture images of reflected IR light and usethese IR light images to detect anchor points in any kind ofenvironment, even in low light environments.

As shown in FIG. 13B, the HMD is able to detect a vast array ofdifferent anchor points 1315. These anchor points 1315 are symbolicallyidentified through the use of the dark circles. In some scenarios, theseanchor points are fixed physical features, such as corners, that areuseful for determining the HMD's position and orientation in relation toits surrounding environment. As will be discussed later, the HMD mayidentify any number of anchor points.

Attention will now be directed to FIG. 14, which illustrates a HMD thatis similar to the HMD 200 from FIG. 2. As a result, the commoncomponents (e.g., the left and right cameras) will not be relabeled.This HMD is specially configured to perform movement detection (e.g.,head tracking, hand tracking, etc.) in low light environments.

As discussed earlier, the HMD includes at least two cameras (though morethan two may be used), both of which are configured to obtain one ormore visible light images of the low light environment as well as one ormore IR light images of the low light environment. Notably, thesevisible light and IR light images may be used to track movements of theHMD. Of course, a single image may capture both visible light and IRlight.

In addition to the cameras, the HMD also includes at least one flood IRlight illuminator. In the scenario shown in FIG. 14, the HMD includes aleft flood IR illuminator 1400 and a right flood IR light illuminator1405. Therefore, in this example scenario, two flood IR lightilluminators are mounted on the HMD, each of which is emitting adifferent flood of IR light. Of course, it will be appreciated that theHMD may include any number of flood IR light illuminators. For example,the HMD may include 1, 2, 3, 4, 5, 6, 7, 8, or any number of flood IRlight illuminators.

These flood IR light illuminators may be example implementations of theflood IR illuminator 1305 from FIG. 13A. Specifically, these flood IRilluminators 1400 and 1405 are configured to emit a flood of IR lightthat spans an illumination area. For example, the left flood IRilluminator 1400 is emitting the flood IR illumination 1410, and theright flood IR illuminator 1405 is emitting the flood IR illumination1415.

As shown, both of the flood IR illuminators 1400 and 1405 are mounted onthe HMD. In some implementations, the left flood IR light illuminator1400 is mounted on the HMD proximately to the left camera, and the rightflood IR light illuminator 1405 is mounted on the HMD proximately to theright camera. Similar to that which was discussed for the dot-patternilluminator, however, the left and right flood IR light illuminators1400 and 1405 may be mounted anywhere on the HMD and in any orientation.

The IR illuminator 1400 is mounted on the HMD relative to the cameras sothat the flood IR illumination 1410 overlaps at least some, andpreferably a majority, of the left camera's field of view 1420.Similarly, the IR illuminator 1405 is mounted on the HMD relative to thecameras so that the flood IR illumination 1415 overlaps at least some,and preferably a majority, of the right camera's field of view 1425.This overlap is illustrated by the overlapping regions 1430 and 1435. Inthis manner, some, and preferably most (or all), of the cameras' fieldsof view will be illuminated with flood IR light. This configuration isadvantageous because it enables the HMD to use its cameras to identifyanchor points within any kind of environment, even low lightenvironments. Of course, the aforementioned resulting overlap may occurin different percentages. For example, the flood IR illuminations 1410and 1415 may overlap the fields of view 1420 and 1425, respectively, by20%, 25%, 30%, 40%, 50%, 60%, 70%, 80%, 90%, or even 100%.

In some circumstances, the flood IR light illuminators 1400 and 1405always emit their respective flood of IR light while the HMD is activelygenerating a mixed-reality environment. In other circumstances, theflood IR light illuminators 1400 and 1405 are triggered to turn on onlywhen a particular condition occurs.

Examples of triggering conditions include, but are not limited to, (1) adetection (either by the HMD's cameras or by an ambient light sensor ofthe HMD) that the ambient light intensity of the surrounding environmenthas fallen below a threshold intensity value (e.g., 40 lux) and/or (2) adetermination that the number of detectable/detected anchor points isbelow a preselected number of anchor points (e.g., 6, 7, 8, 9, 10, 15,20, 25, 30, 35, 40, 45, 50, 55, 60, 90, 100, 200, etc. anchor points).

Additionally, the flood IR light illuminators 1400 and 1405 may operatein a pulsed mode, similar to that which was described above.Accordingly, by projecting a flood of IR light into the HMD'senvironment, the HMD will be able to better determine its positionwithin the environment, and thus better track movements within thatenvironment.

Attention will now be directed to FIGS. 15 through 17, which illustrateexample computer system components and methods that may be used toperform head tracking with the use of flood IR illuminators in low lightenvironments.

FIG. 15 illustrates various different head tracking computer systemcomponents 1500 that may be included in any of the HMDs discussed thusfar. In particular, the components include cameras 1505 (e.g., thecameras 205 and 210 from FIG. 2), one or more flood illuminators 1510(e.g., the left flood IR illuminator 1400 and the right flood IRilluminator 1405 from FIG. 14), and a head tracker 1515.

The illustrated head tracker 1515 is configured, in some instances, toselectively operate in a low light mode 1520 and a non-low light mode1525. The low light mode 1520 is used when the HMD is being utilized ina low light environment. The non-low light mode 1525 is used when theHMD is being utilized in bright light environments. The HMD dynamicallyadjusts modes and controls the timing for turning on/off the flood IRlight illuminators based on detected light conditions.

The HMD system components also include one or more processors 1530 andstorage 1535. The one or more processors 1530 may be speciallyconfigured to operate as the head tracker 1515 and to execute storedcomputer-executable instructions stored in the storage 1535 forimplementing the acts described herein.

FIG. 16 illustrates a flowchart 1600 of various acts associated withmethods for tracking movements of a HMD in any kind of environment,including low light environments. The illustrated acts may be performedby the head tracking computer system components 1500, which may beincluded in a HMD.

The first illustrated act is an act of emitting a flood of IR light froma flood IR light illuminator mounted on the HMD (act 1605). This act maybe performed by the flood illuminators 1510 from FIG. 15, for example.

Next, reflected IR light is detected (act 1610). This act may beperformed by the cameras 1505 from FIG. 15, for example. In addition todetecting the reflected IR light, visible light is also detected usingthe cameras 1505. Therefore, both visible light and IR light aredetected. In this manner, the visible and reflected IR light is detectedby the HMD's head-tracking stereo camera pair. Based on the detectedreflected IR light and/or the detected reflected visible light, aposition of the HMD is determined (act 1615). This act may be performedby the head tracker 1515, for example. Detecting the position of the HMDmay be performed in the manner described above in reference to theinside-out camera detection techniques.

Similar to the IR dot-pattern illuminators that were discussed earlier,the flood IR light illuminators may also be pulsed. When this occurs,the pulsing of the flood IR light illuminators may be synchronized tocoincide with the cameras' exposure times. In other words, the camerasmay also be pulsed. Pulsing the flood IR light illuminators may beadvantageous for reducing illumination thermal loads and batteryconsumption.

FIG. 17 illustrates another related flowchart 1700 of various acts fortracking a position of a HMD within a surrounding environment.Specifically, method 1700 advantageously demonstrates how theillumination intensity of a flood IR light illuminator can bedynamically adjusted to ensure that a sufficient number of anchor pointsare detectable within an IR light image.

As shown, a flood of IR light is initially emitted from a flood IR lightilluminator (act 1705). This act may be performed by the floodilluminators 1510 from FIG. 15, for example. Here, the flood of IR lightmay be emitted at an initial illumination intensity into the surroundingenvironment. Such an operation results in reflected IR light beingreflected from one or more objects in the surrounding environment.

Next, an image that includes digital content corresponding to thereflected IR light (and/or reflected visible light) is captured, orrather obtained, using the HMD's camera system (e.g., the stereo camerapair) (act 1710). This act may be performed by the cameras 1505 fromFIG. 15, for example. In some instances, multiple images are obtainedand then a selected subset of images are be used (e.g., image(s) havingthe least relative amount of blurring).

Subsequently, the HMD identifies a selected quantity of anchor pointsfrom within the captured/selected image(s) (act 1715). This act may beperformed by the head tracker 1515 from FIG. 15, for example.

Finally, the illumination intensity is subsequently modified (act 1720).The modification to the illumination intensity of the floodilluminator(s) 1510 can be controlled by the head tracker 1515. Such amodification occurs by increasing or decreasing a power supplied to atleast one flood IR light illuminator. Increasing or decreasing thispower alters the amount of reflected IR light that is detectable by thecameras. In a corresponding manner, the modification can also alter thequantity of anchor points that are detectable in a subsequent imagecaptured by the camera system. This modification may iteratively anddynamically occur until the quantity of anchor points that aredetectable satisfies a predetermined threshold quantity of anchor points(e.g., 6, 7, 8, 9, 10, 15, 20, 30, 50, 100, 200, etc.).

The process of identifying the anchor points may additionally include(1) measuring a mean illumination intensity over a set of pixels in thecaptured image and then (2) indexing the set of pixels in the capturedimage to a particular anchor point when the mean illumination intensityfor that set of pixels reaches a predefined value.

In some instances, machine learning image segmentation is also performedto enhance/supplement/replace the mapping of anchor points from capturedimages.

Additionally, or alternatively, the process of adjusting theillumination intensity to identify anchor points may include enforcing apredefined signal to noise characteristic for any image pixels that aredetermined to correspond to that anchor point. To illustrate, for everyintensity level, it is possible to measure the noise level by computingthe variance of intensity values within an image patch. The intensitycan then be adjusted until the variance equals a predefined value. Asdescribed below, the depth map may also be analyzed to help with theprocess of determining which pixels correspond with which objects.

To illustrate, the anchor points may be identified by factoring in thedepth of an object. Even more particularly, it is generally known thatlight decays with the square of the distance. This means that the HMDmay use 4 times the illumination strength to illuminate an object's areaat a distance of 2 meters as compared to areas at a depth of 1 meter.Depth detection provides information about the current object's depthfor every frame. If the HMD is moved closer to the object, the HMD canthen reduce the illumination power. In the case of moving away from theobject, then the HMD may increase the illumination strength. As aresult, there are many different processes for illuminating andidentifying anchor points and for determining when and how much todynamically modify the light intensity.

Therefore, by incorporating the use of one or more flood IR lightilluminators, the HMD is able to better track movements within any kindof environment, including low light environments.

Hybrid System for Determining Depth and Tracking Movement

The following disclosure provides descriptions of hybrid systems thatare configured to perform depth detection and tracking with HMD systemsin various environments, even low light environments. Such hybridsystems are beneficially used for hand tracking purposes. Previousembodiments focused on generating a depth map that corresponds to alarge surrounding HMD environment. In some of the following embodiments,however, only a limited depth map is used for performing hand tracking.

Turning first to FIG. 18, a HMD is illustrated that includes one or moreflood IR light illuminators, one or more cameras, and one or more IRdot-pattern illuminators. Because these components are similar to thecomponents shown in FIGS. 7 and 14, the common components have not beenrelabeled.

As shown, the IR dot-pattern illuminator 1830 is emitting an IRdot-pattern illumination 1800 which adds texture to the HMD'senvironment. Additionally, the left flood IR light illuminator 1840 isemitting a flood IR illumination 1805, and the right flood IR lightilluminator 1842 is emitting a flood of IR illumination 1810. The floodIR illuminations 1805, 1810 overlap the fields of view 1850 of the HMDcameras 1820, as previously discussed. Notably, these cameras 1820 areconfigured to detect both visible light and IR light.

In this embodiment, the IR dot-pattern illuminator 1830 is emitting theIR dot-pattern illumination 1800 according to a particular frequency(e.g., the frequencies that were discussed earlier). Additionally, asalso discussed, the cameras 1820 may be synchronized with operationalfrequency of the IR dot pattern illuminator 1830.

In this embodiment, the flood IR light illuminators 1840 and 1842 areconfigured to emit a flood of IR light to illuminate the mixed-realityscene in an effort to improve movement detection, regardless of whetherthat movement is the HMD's movement or a hand movement. Here, the floodIR light illuminators 1840 and 1842 emit the flood of IR light accordingto a different frequency than the frequency of the IR dot patternilluminator 1830. The cameras are also synchronized to capture images atthe operational frequency of the IR pattern illuminator 1830, so as toobtain images for head tracking purposes. A third camera frequency isalso used, in some instances, for capturing images used for handtracking, as will be described in more detail later.

With that said, attention will now be directed to FIG. 19 whichillustrates hybrid head (and also hand) tracking and depth detectorcomputer system components 1900. In particular, these components includecameras 1905, flood illuminator(s) 1910, a head tracker 1915, which mayinclude at least a low light mode 1920, an IR dot-pattern illuminator1925, and a depth detector 1930. The components also include one or moreprocessors 1935 and storage 1940. The one or more processors 1935 may bespecially configured as the head tracker 1915 and the depth detector1930 and to executed stored computer-executable instructions stored inthe storage for implementing the disclosed methods.

Attention will now be directed to FIG. 20 which illustrates a flowchart2000 of various acts associated with performing camera-based tracking ofmovements of a head-mounted device (HMD) in a low light environment inconjunction with determining object depth. These acts are performed by aHMD that includes the hybrid head tracking and depth detector computersystem components 1900 from FIG. 19, for example.

As shown, the illustrated acts are separated into two columns. Thisformat demonstrates that at least some of the acts are not temporallydependent on all other acts. This format is similar to the formatdiscussed in connection with FIG. 12.

Focusing first on the left column (i.e. the column labeled “FrequencyA”), a number of acts are performed at a first selected frequency (e.g.,the frequencies suitable for performing tracking). The first act is anact for emitting flood IR light with a flood IR light illuminator (act2005). This act may be performed by the flood illuminators 1910 fromFIG. 19, for example.

With regard to the act of emitting the flood of IR light, it will beappreciated that the IR light can be emitted at different intensities,depending on the mode of operation, as discussed earlier. For instance,the intensity of flood IR light (or even the IR dot-pattern light) maybe dynamically adjusted to accommodate head tracking scenarios and handtracking scenarios. For instance, because hands are often quite close tothe HMD, the HMD will reduce the intensity level of the IR light whenperforming hand tracking. In contrast, when it is determined the systemis performing head tracking, the HMD will increase the intensity levelof the IR light to more effectively illuminate objects in thesurrounding environment (further than the user's hand).

Next, the head tracking stereo camera pair obtains one or more images ofreflected IR light and/or reflected visible light (act 2010). Then, oncethe images are obtained, they are used to track the movements of the HMD(act 2015), such as by detecting the relative position/orientation ofdetected objects relative to detected anchor points (as describedabove). Notably, these operations may be performed by the head trackingcameras of a HMD even when the HMD is operating in a low lightenvironment, by utilizing the IR light reflections.

Focusing now on the right column (i.e. the column labeled “FrequencyB”), there are a number of method acts that may be performed at adifferent frequency than Frequency A. At this point, it is worthwhile tonote that although the method acts are shown as occurring at aset/particular frequency, the method acts may also be performed atvariable frequencies (asynchronously triggered) in response to variousdetectable conditions. For example, if the HMD determines that the lowlight environment includes objects having low or no texture, then theHMD may asynchronously trigger its IR dot-pattern light illuminator toemit the dot-pattern illumination. Regardless of whether the acts aretriggered asynchronously or at a fixed period in accordance with anestablished frequency, the HMD is able to cause its IR dot-patternilluminator to emit an IR dot-pattern illumination onto at least a partof the low light environment (act 2020). This act may be performed bythe IR dot-pattern illuminator 1925, for example.

As further illustrated, the head tracking stereo camera pair obtain oneor more images that include digital content corresponding to thereflected IR dot-pattern light and/or the reflected visible light (act2025).

Finally, the one or more images of the reflected IR dot-pattern lightare used to construct depth map and one or more geometric surfaces inthe surrounding environment (act 2030). As one example, the geometricsurface may be for the user's hand while in another example thegeometric surface may be for a textureless/smooth table. This may beperformed, for example, by the depth detector 1930.

In some embodiments, additional acts are performed at yet anotherfrequency. For example, the HMD may also obtain images used to monitorthe user's hand movements. Initially, the HMD passively performs theseoperations at a low frequency (e.g., 3 frames per second) until suchtime as a hand is eventually identified within an image. When the user'shand is finally detected in an image, then the HMD dynamically changesthe IR light illuminator and/or camera processes to be performed at ahigher frequency (e.g., anywhere between 30 frames per second to 60frames per second). By dynamically increasing the operationalfrequencies in this manner, the HMD is able to better adjust for andtrack the movements of a user's hand during use of a HMD.

Notably, the hand-tracking images are often obtained using the headtracking cameras and the flood IR light illuminators. Consequently, theflood IR illuminators may be multi-purposed to perform both headtracking and hand tracking. To determine whether a hand is present in animage, the HMD has been configured/trained (e.g., through machinelearning) to detect hands and to perform hand segmentation on the imagesthat were captured using the flood of IR light. In alternativesituations, the HMD may use the depth map that was generated using theIR dot-pattern illuminators to perform object segmentation and/or objectmatching.

In some embodiments, the HMD actively tracks the user's hand at thehigher/modified frequency until the user's hand is no longer detectablein the images. Once the hand is no longer detectable, then the HMD mayreturn to the lower frequency (e.g., 3 frames per second).

It will be appreciated from this disclosure, the HMD systems describedherein may be used to perform a number of different tracking and depthdetection operations at different operational frequencies. For example,at one frequency, the HMD may perform head tracking. At a differentfrequency, the HMD may perform hand tracking. At yet another frequency,the HMD may perform depth detection. Of course, these frequencies aredynamically adjustable, based on detected environmental conditionsand/or modes of operation. Furthermore, the HMD is able to dynamicallyadjust the intensity of IR light emitted by the flood IR lightilluminators and/or the IR dot-pattern illuminators to account for thesedifferent modes of operation.

It will also be appreciated that the HMD may support any number ofcameras. For instance, to perform head and/or hand tracking, the HMD mayuse/include only a single camera (i.e. mono camera implementation).Alternatively, as discussed earlier, the HMD may include any number ofcameras (one, or more than one).

In some embodiments, there are multiple sets of stereo camera pairs. Insome embodiments, depth may also be obtained using a single camera. Forinstance, depth can be measured from parallax that occurs when a singlecamera moves. Such computations provide a relative depth as opposed toan absolution depth. It is possible, for instance, to determine that thedepth of a pixel is smaller than the depth of another pixel, and tothereby provide the relative depth determination. Higher degrees/amountsof parallax might be observed because the camera undergoes a largemotion and the point has large depth. Alternatively, higherdegrees/amounts of parallax might be observed if the point has smalldepth and there was only a small motion. Because of this ambiguity, theresults are accurate up to a scale factor and hence the HMD obtains adifferent output compared to the stereo approach for determining depth.

Example Computer System

Having just described the various features and functionalities of someof the disclosed embodiments, the focus will now be directed to FIG. 21which illustrates an example computer system 2100 that may be used tofacilitate the operations described herein. In particular, this computersystem 2100 may be in the form of the HMDs that were described earlier.

In fact, the computer system 2100 may take various different forms. Forexample, in FIG. 21, the computer system 2100 is embodied as a HMD.Although the computer system 2100 may be embodied as a HMD, the computersystem 2100 may also be a distributed system that includes one or moreconnected computing components/devices that are in communication withthe HMD. Accordingly, the computer system 2100 may be embodied in anyform and is not limited strictly to the depiction illustrated in FIG.21. By way of example, the computer system 2100 may include a desktopcomputer, a laptop, a tablet, a mobile phone, server, data center and/orany other computer system.

In its most basic configuration, the computer system 2100 includesvarious different components. For example, FIG. 21 shows that computersystem 2100 includes at least one hardware processing unit 2105 (aka a“processor”), input/output (I/O) interfaces 2110, graphics renderingengines 2115, one or more sensors 2120, and storage 2125. Regarding thehardware processing unit 2105, this may be an example implementation ofthe processors 1025 from FIG. 10, the processors 1530 from FIG. 15, andthe processors 1935 from FIG. 19. More detail on the hardware processingunit 2105 will be presented momentarily.

The storage 2125 may be physical system memory, which may be volatile,non-volatile, or some combination of the two. The term “memory” may alsobe used herein to refer to non-volatile mass storage such as physicalstorage media. If the computer system 2100 is distributed, theprocessing, memory, and/or storage capability may be distributed aswell. As used herein, the term “executable module,” “executablecomponent,” or even “component” can refer to software objects, routines,or methods that may be executed on the computer system 2100. Thedifferent components, modules, engines, and services described hereinmay be implemented as objects or processors that execute on the computersystem 2100 (e.g. as separate threads).

The disclosed embodiments may comprise or utilize a special-purpose orgeneral-purpose computer including computer hardware, such as, forexample, one or more processors (such the hardware processing unit 2105)and system memory (such as storage 2125), as discussed in greater detailbelow. Embodiments also include physical and other computer-readablemedia for carrying or storing computer-executable instructions and/ordata structures. Such computer-readable media can be any available mediathat can be accessed by a general-purpose or special-purpose computersystem. Computer-readable media that store computer-executableinstructions in the form of data are physical computer storage media.Computer-readable media that carry computer-executable instructions aretransmission media. Thus, by way of example and not limitation, thecurrent embodiments can comprise at least two distinctly different kindsof computer-readable media: computer storage media and transmissionmedia.

Computer storage media are hardware storage devices, such as RAM, ROM,EEPROM, CD-ROM, solid state drives (SSDs) that are based on RAM, Flashmemory, phase-change memory (PCM), or other types of memory, or otheroptical disk storage, magnetic disk storage or other magnetic storagedevices, or any other medium that can be used to store desired programcode means in the form of computer-executable instructions, data, ordata structures and that can be accessed by a general-purpose orspecial-purpose computer.

The computer system 2100 may also be connected (via a wired or wirelessconnection) to external sensors 2130 (e.g., one or more remote cameras,accelerometers, gyroscopes, acoustic sensors, magnetometers, etc.).Further, the computer system 2100 may also be connected through one ormore wired or wireless networks 2135 to remote systems(s) 2140 that areconfigured to perform any of the processing described with regard tocomputer system 2100.

During use, a user of the computer system 2100 is able to perceiveinformation (e.g., a mixed-reality environment) through a display screenthat is included among the I/O interface(s) 2110 and that is visible tothe user. The I/O interface(s) 2110 and sensors 2120/2130 also includegesture detection devices, eye trackers, and/or other movement detectingcomponents (e.g., cameras, gyroscopes, accelerometers, magnetometers,acoustic sensors, global positioning systems (“GPS”), etc.) that areable to detect positioning and movement of one or more real-worldobjects, such as a user's hand, a stylus, and/or any other object(s)that the user may interact with while being immersed in the scene.

The graphics rendering engine 2115 is configured, with the hardwareprocessing unit 2105, to render one or more virtual objects within thescene. As a result, the virtual objects accurately move in response to amovement of the user and/or in response to user input as the userinteracts within the virtual scene.

A “network,” like the network 2135 shown in FIG. 21, is defined as oneor more data links and/or data switches that enable the transport ofelectronic data between computer systems, modules, and/or otherelectronic devices. When information is transferred, or provided, over anetwork (either hardwired, wireless, or a combination of hardwired andwireless) to a computer, the computer properly views the connection as atransmission medium. The computer system 2100 will include one or morecommunication channels that are used to communicate with the network2135. Transmissions media include a network that can be used to carrydata or desired program code means in the form of computer-executableinstructions or in the form of data structures. Further, thesecomputer-executable instructions can be accessed by a general-purpose orspecial-purpose computer. Combinations of the above should also beincluded within the scope of computer-readable media.

Upon reaching various computer system components, program code means inthe form of computer-executable instructions or data structures can betransferred automatically from transmission media to computer storagemedia (or vice versa). For example, computer-executable instructions ordata structures received over a network or data link can be buffered inRAM within a network interface module (e.g., a network interface card or“NIC”) and then eventually transferred to computer system RAM and/or toless volatile computer storage media at a computer system. Thus, itshould be understood that computer storage media can be included incomputer system components that also (or even primarily) utilizetransmission media.

Computer-executable (or computer-interpretable) instructions comprise,for example, instructions that cause a general-purpose computer,special-purpose computer, or special-purpose processing device toperform a certain function or group of functions. Thecomputer-executable instructions may be, for example, binaries,intermediate format instructions such as assembly language, or evensource code. Although the subject matter has been described in languagespecific to structural features and/or methodological acts, it is to beunderstood that the subject matter defined in the appended claims is notnecessarily limited to the described features or acts described above.Rather, the described features and acts are disclosed as example formsof implementing the claims.

Those skilled in the art will appreciate that the embodiments may bepracticed in network computing environments with many types of computersystem configurations, including personal computers, desktop computers,laptop computers, message processors, hand-held devices, multi-processorsystems, microprocessor-based or programmable consumer electronics,network PCs, minicomputers, mainframe computers, mobile telephones,PDAs, pagers, routers, switches, and the like. The embodiments may alsobe practiced in distributed system environments where local and remotecomputer systems that are linked (either by hardwired data links,wireless data links, or by a combination of hardwired and wireless datalinks) through a network each perform tasks (e.g. cloud computing, cloudservices and the like). In a distributed system environment, programmodules may be located in both local and remote memory storage devices.

Additionally or alternatively, the functionality described herein can beperformed, at least in part, by one or more hardware logic components(e.g., the hardware processing unit 2105). For example, and withoutlimitation, illustrative types of hardware logic components that can beused include Field-Programmable Gate Arrays (FPGAs), Program-Specific orApplication-Specific Integrated Circuits (ASICs), Program-SpecificStandard Products (ASSPs), System-On-A-Chip Systems (SOCs), ComplexProgrammable Logic Devices (CPLDs), Central Processing Units (CPUs), andother types of programmable hardware.

The disclosed embodiments provide various advantages over traditionalHMD systems. Some of these advantages include providing a more robustand accurate depth determination for mixed-reality environments,particularly low light environments. Additionally, some of theseadvantages include the ability to track movement (e.g., head movement,hand movement, etc.) in any kind of environment, even low lightenvironments. Furthermore, by repurposing existing hardware components,such as the head tracking cameras to additionally perform depthdetection, the disclosed embodiments can reduce/simplify the costs,power consumption and form factor of the HMD systems.

The present invention may be embodied in other specific forms withoutdeparting from its spirit or characteristics. The described embodimentsare to be considered in all respects only as illustrative and notrestrictive. The scope of the invention is, therefore, indicated by theappended claims rather than by the foregoing description. All changeswhich come within the meaning and range of equivalency of the claims areto be embraced within their scope.

1-7. (canceled)
 8. A method for camera-based tracking of movements of ahead-mounted device (HMD) in a low light environment in conjunction withdetermining object depth, the method being performed by the HMD, themethod comprising: at a first selected frequency, performing at leastthe following: causing a flood infrared (IR) light illuminator to emit aflood of IR light into the low light environment; causing a headtracking stereo camera pair to obtain one or more images of reflected IRlight resulting from the flood of IR light and reflected visible lightreflecting off of one or more objects in the low light environment,wherein the head tracking stereo camera pair includes a first camera anda second camera; and using the one or more images to track the movementsof the HMD; and at a second selected frequency, performing at least thefollowing: causing a dot-pattern IR illuminator to emit a dot-pattern IRillumination onto at least a part of the low light environment; causingthe head tracking stereo camera pair to obtain one or more images ofreflected dot-pattern IR light resulting from the dot-patternillumination; and using the one or more images of the reflecteddot-pattern IR light and the reflected visible light to construct ageometric surface for at least the part of the low light environment. 9.The method of claim 8, wherein the HMD is a virtual reality system. 10.The method of claim 8, wherein the HMD is an augmented reality system.11. The method of claim 8, wherein the first selected frequency is atleast 15 frames per second.
 12. The method of claim 8, wherein thesecond selected frequency is at least 0.5 frames per second.
 13. Themethod of claim 8, wherein the method further includes one or more of:segmenting a hand of a user within the one or more images; or using agenerated 3D depth map to calculate a size and/or a pose of the user'shand.
 14. The method of claim 8, wherein the one or more images of thereflected dot-pattern IR light include an image of a hand of a user, andwherein the method further includes: after identifying the user's handin the one or more images of the reflected dot-pattern IR light,adjusting the second selected frequency to a higher frequency; and tracka movement of the user's hand at the higher frequency.
 15. The method ofclaim 14, wherein after determining that the user's hand is no longer inthe one or more images of the reflected dot-pattern IR light, adjustingthe second frequency to a lower frequency.
 16. A head-mounted device(HMD) comprising: one or more processors; and one or morecomputer-readable hardware storage devices having stored thereoncomputer executable instructions that are executable by the one or moreprocessors to cause the HMD to perform a method for tracking movementsof the HMD in a low light environment in conjunction with determiningobject depth, wherein the method comprises: at a first selectedfrequency, performing at least the following: causing a flood infrared(IR) light illuminator to emit a flood of IR light into the low lightenvironment; causing a head tracking stereo camera pair to obtain one ormore images of reflected IR light resulting from the flood of IR lightand reflected visible light reflecting off of one or more objects in thelow light environment, wherein the head tracking stereo camera pairincludes a first camera and a second camera; and using the one or moreimages to track the movements of the HMD; and at a second selectedfrequency, performing at least the following: causing a dot-pattern IRilluminator to emit a dot-pattern IR illumination onto at least a partof the low light environment; causing the head tracking stereo camerapair to capture one or more images of reflected dot-pattern IR lightresulting from the dot-pattern illumination and reflected visible light;and using the one or more images of the reflected dot-pattern IR lightand the reflected visible light to construct a geometric surface for atleast the part of the low light environment.
 17. The HMD of claim 16,wherein the dot-pattern IR illuminator is positioned on the HMD closerto the first camera than to the second camera.
 18. The HMD of claim 16,wherein the HMD includes a second flood IR light illuminator.
 19. TheHMD of claim 18, wherein the flood IR light illuminator is mounted onthe HMD proximately to the first camera and the second flood IR lightilluminator is mounted on the HMD proximately to the second camera. 20.The HMD of claim 16, wherein the method further includes: determiningthat at least the part of the low light environment has low texture; andtriggering the dot-pattern IR light illuminator to emit the dot-patternillumination onto at least the part of the low light environment inresponse to determining that at least the part of the low lightenvironment has low texture.
 21. One or more hardware storage device(s)having stored thereon computer executable instructions that areexecutable by one or more processor(s) to cause a head-mounted device(HMD) to track movements of the HMD in a low light environment inconjunction with determining object depth by causing the HMD to at leastperform the following: at a first selected frequency, performing atleast the following: causing a flood infrared (IR) light illuminator toemit a flood of IR light into the low light environment; causing a headtracking stereo camera pair to obtain one or more images of reflected IRlight resulting from the flood of IR light and reflected visible lightreflecting off of one or more objects in the low light environment,wherein the head tracking stereo camera pair includes a first camera anda second camera; and using the one or more images to track the movementsof the HMD; and at a second selected frequency, performing at least thefollowing: causing a dot-pattern IR illuminator to emit a dot-pattern IRillumination onto at least a part of the low light environment; causingthe head tracking stereo camera pair to capture one or more images ofreflected dot-pattern IR light resulting from the dot-patternillumination and reflected visible light; and using the one or moreimages of the reflected dot-pattern IR light and the reflected visiblelight to construct a geometric surface for at least the part of the lowlight environment.
 22. The one or more hardware storage device(s) ofclaim 21, wherein the dot-pattern IR illuminator is positioned on theHMD closer to the first camera than to the second camera.
 23. The one ormore hardware storage device(s) of claim 21, wherein the HMD includes asecond flood IR light illuminator.
 24. The one or more hardware storagedevice(s) of claim 23, wherein the flood IR light illuminator is mountedon the HMD proximately to the first camera and the second flood IR lightilluminator is mounted on the HMD proximately to the second camera. 25.The one or more hardware storage device(s) of claim 21, whereinexecution of the computer-executable instructions further causes the HMDto: determine that at least the part of the low light environment haslow texture; and trigger the dot-pattern IR light illuminator to emitthe dot-pattern illumination onto at least the part of the low lightenvironment in response to determining that at least the part of the lowlight environment has low texture.
 26. The one or more hardware storagedevice(s) of claim 21, wherein the first selected frequency is at least15 frames per second.
 27. The one or more hardware storage device(s) ofclaim 21, wherein the second selected frequency is at least 0.5 framesper second.